Statistical Sparse Distributed Memory Prediction with Kanerva's

A new viewpoint of the processing: performed by Kanerva's sparse distributed memory (SDM) is presented. In conditions of nearor overcapacity, where the associative-memory behavior of the model breaks down, the processing performed by the model can be interpreted as that of a statistical predictor. Mathematical results are presented which serve as the framework for a new statistical viewpoint of sparse distributed memory and for which the standard formulation of SDM is a special case. This viewpoint suggests possible enhancements to the SDM model, including a procedure for improving the predictiveness of the system based on Holland's work with 'Genetic Algorithms', and a method for improving the capacity of SDM even when used as an associative memory. OVERVIEW This work is the result of studies involving two seemingly separate topics that proved to share a common framework. The first topic, statistical prediction, is the task of associating exlremely large perceptual state vectors with future events. The second topic, over-capacity in Kanerva's sparse distributed memory (SDM), is a study of the computation done in an SDM when presented with many more associations than its stated capacity. I propose that in conditions of over-capacity, where the associative-memory behavior of an SDM bre_ks-dowh-,-th-e-p_r0cessihg performed by the SDM can be used for statistical prediction. A mathematical study of the prediction problem suggests a variant of the standard SDM architectur e. This variant not only behaves as a statistical predictor when the SDM is filled beyond capacity but is shown to double the capacity of an SDM when used as an associative memory. THE PREDICTION PROBLEM The earliest living creatures had an ability, albeit limited, to perceive the world through crude senses. This ability allowed them to react to changing conditions in theenvironment; for example, to move towards (or away from) light sources. As riervous systems developed, learning was possible; if food appeared simultaneously with some other perception, perhaps some odor, a creature could learn to associate that smell with food. As the creatures evolved further, a more rewarding type of learning was possible. Some perceptions, such as the perception of pain or the discovery of food, are very important to an animal. However, by the time the perception occurs, damage may already be done, or an opportunity for gain missed. If a creature could learn to associate eta-rent perceptions with future ones, it would have a much better chance to do something about it before damage occurs. This is the prediction problem. The difficulty of the prediction problem is in the extremely large number of possible sensory inputs. For example, a simple animal might have the equivalent of 1000 bits of sensory data at a given time; in this case, the number of possible inputs is greater than the number of atoms in the known universe! In essence, it is an enormous search problem: a living creature must find the subregions of the perceptual space which correlate with the features of interest. Most of the gigantic perceptual space will be uncorrelated, and hence uninteresting. THE OVERCAPACITY PROBLEM An associative memory is a memory that can recall data when addressed 'close-to' an address where data were prevlously stored. A number of designs for associative memories have been proposed, such as Hopfield netwt_rks (Hopfield, 1986) or the nearest-neighbor associative memory of Baum, Moody, and Wilczek (1987). Memory-related standards such as capacity are usually selected to judge the relative performance of different models. Performance is severely degraded when these memories are filled beyond capacity. Kanerva's sparse distributed memory is an associative memory model developed from the mathematics of high-dimensional spaces (Kanerva, 1988) and is related to the work of David Mart (1969) and James Albus (1971) on the cerebellum of the brain. (For a detailed comparison of SDM to random-access memory, to the cerebellum, and to neural-networks, see (Rogers, 1988b)). Like other associative memory models, it exhibits non-memory behavior when nearor overcapacity. Studies of capacity are often over-simplified by the common assumption of uncorrelated random addresses and data. The capacity of some of these memories, including SDM, is degraded if the memory is presented with correlated addresses and data. Such correlations are likely if the addresses and data are from a real-world source. Thus, understanding the over-capacity behavior of an SDM may lead to better procedures for storing correlated data in an associative memory. SPARSE DISTRIBUTED MEMORY Sparse distributed memory can be best illustrated as a variant of random-access memory (RAM). The structure of a twelve-location SDM with ten-bit addresses and ten-bit data is shown in figure 1. (Kanerva, 1988) Reference Address InputData I o,ot0,01,01 Iol ol ol 1111,1ol ,101,1 ,,0,,00,11 ,0,010,010 000001,,,0 001,0,,00, 1011,0,,00 .4_ 00,0,0,1,, ,,0,,0,10, 0,000001,0 0,,0,0,001 ,0,10,0,10 ,,000,0,,, "_ 111,11001, "_" Radius 2] Dist 4 8 3 6 7 7 6 2 8 3 3 5 ;elect •-_ 0 "_